Methods and systems for obstacle detection using structured light

ABSTRACT

An obstacle detector for a mobile robot while the robot is in motion is disclosed. The detector preferably includes at least one light source configured to project pulsed light in the path of the robot; a visual sensor for capturing a plurality of images of light reflected from the path of the robot; a processing unit configured to extract the reflections from the images; and an obstacle detection unit configured to detect an obstacle in the path of the robot based on the extracted reflections. In the preferred embodiment, the reflections of the projected light are extracted by subtracting pairs of images in which each pair includes a first image captured with the at least one light source on and a second image captured with the at least one light source off, and then combining images of two or more extracted reflections to suppress the background.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

Any and all applications for which a foreign or domestic priority claimis identified in the Application Data Sheet as filed with the presentapplication are hereby incorporated by reference under 37 CFR 1.57.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This disclosure presents a system and method through which a mobiledevice, such as a robot, can detect obstacles while navigating in anenvironment using a laser pattern to augment the visible field of avisual sensor and measure the distance to objects.

2. Description of the Related Art

To create a robust collision detection system, an algorithmic approachusing only a video feed is unlikely to detect all near-field objectswith a reliability preferred for robotic navigation.

Automatic obstacle detection is a useful functionality for a mobiledevice that navigates in an environment. For fully autonomous devices,automatic obstacle detection enables the device to move within anenvironment without damaging objects and without getting caught, pinned,jammed, or trapped. For a semi-autonomous device, such as tele-operatedvehicle, automatic obstacle detection can facilitate the tele-operatedcontrol of the vehicle by either alerting a human operator to thepresence of obstacles or by adapting the operator's commands in order toavoid collisions with detected obstacles.

There are several known methods for automatic obstacle detection. Oneconventional method for obstacle detection uses a laser range finder. Ina laser range finder, a laser beam is pulsed, and the time required toreceive the reflection of the pulse is proportional to the distance toan obstacle. The laser and detector are typically mounted on a rotatingplatform so that a scan of the environment can be performed. Typicalscan rates are 10 scans per second, with 360 or more points scanned perrotation. The accuracy is also relatively high, with the estimateddistance to an obstacle typically detected with accuracy better than 1cm and with a range of up to 80 meters. Despite the relatively highquality of the data, the device is relatively expensive, precluding itsuse in many products. Furthermore, the scan is done only in one plane(typically a horizontal plane if the laser range finder is horizontallymounted); thus, the sensor is unable to detect objects that do not liein the scan plane (such as a tabletop).

Another conventional method for obstacle detection uses infrared (IR)sensors. An IR sensor typically comprises an IR light-emitting diode(LED) illumination source with a lens so as to project the light in athin, pencil-like beam, and a photosensitive detector that is alignedwith the illuminator. The detector is typically a one-dimensional (1-D)array, and given the position along the 1-D array where the illuminationis detected, the distance to an obstacle can be calculated. IR sensorstypically have a range on the order of 1 meter. Because IR sensors onlyprovide detection along the pencil-like beam, it is, disadvantageously,relatively difficult to obtain complete coverage of the environment,even when a relatively large number of IR sensors, such as dozens of IRsensors, are used.

Another conventional method for obstacle detection uses ultrasoundsensors. In an ultrasound sensor, an emitter of ultrasonic soundgenerates an ultrasonic sound pulse. The time it takes to receive thereflected echo is proportional to the distance to the obstacle.Ultrasound typically has a range of up to 8 meters, and a singledetector can typically cover a cone of approximately 30 degrees.However, using ultrasound sensors, it is relatively difficult tolocalize an object precisely, to disambiguate between various objects atdifferent distances, and to disambiguate echoes from true objects.

There are several conventional approaches to obstacle detection in whichthe sensor used is a 2-dimensional visual sensor such as that of a CCDcamera, and the techniques used are those of computer vision or machinevision, wherein the image produced is analyzed in one of several ways.One vision-based approach uses stereo vision. In the stereo visionapproach, images of the environment are taken by two or more sensorsamong which the relative positioning is known. By finding pixelcorrespondence among the multiple views of features in the environment,the pixel coordinates can be triangulated to determine the 3-D locationof the detected features. A challenging part of this process isdetermining the correct correspondences of features seen in one imagewith what is seen in other images. This is a relatively computationallyintensive process, which can often produce inaccurate results,particularly when the environment being imaged has relatively littletexture (i.e., features, or points that can easily be identified acrossimages).

Another conventional vision-based approach uses a technique known asstructure from motion. This method is a variant of the stereo visionmethod, in which instead of using images from different imagers that arespatially co-located, images from the same imager taken at differentinstances in time as the mobile device moves through the environment areutilized. The method of detecting corresponding image points and themethod of triangulation are the same as in stereo vision, but theproblem is made more difficult because the spatial co-location of theimager when the images were acquired is, a priori, not known. Even ifthe motion of the mobile device is somehow measured, it is typically notknown with sufficient accuracy to enable accurate triangulation. Thesolution to this problem is typically known in the computer visioncommunity as the “Structure from Motion” problem, and several standardtechniques have been developed to solve it. The overall method is ofcomparable computational cost and complexity as the stereo visionmethod, and the method suffers from the same drawback, in that ittypically does not provide reliable and complete estimates when there isinsufficient texture in the environment.

Another variety of vision-based approaches use structured light. Thismethod overcomes the possibility that there are not enough features inthe environment by using a dedicated light source to generate visiblefeatures in the environment. The light source can be a simple pointsource, such as a laser beam or a focused LED beam, can be a stripe oflight, or can be any other fixed projection pattern. Typically, therelative position of the illumination source to the imager is known, sothat once a point from the illumination pattern is detected, the 3-Dposition of the obstacle on which the illumination pattern point wasdetected can be triangulated. One challenge of this method is toreliably detect the illumination pattern. A typical difficulty is thatthe pattern must remain visible to the imager even in the presence ofother light sources, such as the sun in the case of an outdoorenvironment. There are several known approaches to improve or maximizethe visibility of the pattern.

In one approach for improving or maximizing the visibility of astructured light pattern, the illumination source can be made verybright, so that it is not washed out by other sources. In someapplications this may not be feasible, due to power considerations ordue restrictions on illumination intensity required by eye safetystandards.

In another approach for improving or maximizing the visibility of astructured light pattern, a special optical filter is placed on theimager so as to allow only the frequency of light produced by thepattern illumination source. This will block out a relatively largeportion of the light produced by other sources, which makes the patternillumination source relatively clearly visible in the image. Typically,the use of such a filter produces an image in which the only visibleelement is the illumination pattern. The use of such a filter, however,precludes the use of the imager for other purposes for which otherfrequencies of light are necessary.

In another approach for improving or maximizing the visibility of astructured light pattern, the pattern illumination source can be flashedon very intensely for a very short period of time (typically tens ofmilliseconds or less). Concurrently, the imager is synchronized with theillumination source so as to collect light to form an image only duringthe short interval during which the light source is on. This methodproduces an image where the illumination pattern is clearly visible(again, typically the pattern is the only thing registered in theimage), because during the time the illumination is on, it is of muchhigher intensity than any other source. However, once again, this methodprecludes the use of the imager for other purposes for which otherfrequencies of light are necessary.

In yet another approach for improving or maximizing the visibility of astructured light pattern, the detectability of the pattern can beenhanced by taking two images in rapid succession, one with the patternsource on and another with the pattern source off, and detecting thepattern in the difference of the two images. If the two images are takenin sufficiently rapid succession, then, presumably, the environment,which includes other light sources, has not changed, and so theillumination pattern should be easily detectable as the only differencebetween the two images. This method enables detection of the patternwith a much weaker illumination source than the other methods. However,it works effectively only if there is neither motion of the imager normotion of anything else in the environment. Any moving object (orapparent moving environment if the imager itself is moving) willregister in the difference of images, making it relatively difficult todetect the illumination pattern. Furthermore, many indoor lightingsources connected to alternating current sources experience powerflicker (in the case of fluorescent lights) or fluctuate (in the case ofincandescent lights), typically at 50 Hz or 60 Hz, which also violatesthe assumption that nothing but the pattern illumination source variesbetween the two acquired images.

SUMMARY OF THE INVENTION

The methods and systems for obstacle detection disclosed herein use thestructured light approach, in which novel methods and systems for lightdetection and for sensor placement enable the robust detection ofobstacles in applications where other methods would either not bepermitted (due to eye safety restrictions), or would not functionsatisfactorily.

In one embodiment, the method and system allow detection of a potentialcollision with any part of the forward-looking profile of the mobiledevice. The method and system also do not require the use of a dedicatedimager that is specifically configured for obstacle detection and cannotbe used for any other purpose. Instead, the method and system permitsthe concurrent use of the imager for other purposes, such as objectrecognition, object tracking, or video streaming for surveillance,tele-operation, or video-conferencing.

In one embodiment, the method comprises flashing a laser or LED on andoff; synchronizing the laser with an imager; using a persistence modelto enhance laser signal and remove other sources; optionally, using apseudo-random sequence to further suppress the effects of other lightsources; optionally, triangulating detected point's 3-D position; and,optionally, deploying multiple light sources to fully cover the device'sprofile.

In one embodiment, the method comprises flashing a laser or LED on andoff; synchronizing the laser or LED with an imager; using an opticalbandpass filter in the imager to enhance the laser signal and suppressother light sources and background; using a persistence model to furtherenhance the laser signal and remove other sources; optionally, using apseudo-random sequence to further suppress the effects of other lightsources; and optionally, deploying multiple light sources to fully coverthe device's profile.

In one embodiment, the method comprises any of the systems or methodsdescribed herein together with a tilt sensor or similar measuring deviceto estimate the robot and sensor orientation; and using the sensororientation information to differentiate between genuine obstacles andthe obstruction of field of view due to the robot tilt.

In one embodiment, the invention features an obstacle detector for usein a mobile robot, for example. The detector includes: at least onepulsed light source configured to project light in the path of therobot; a visual sensor for capturing images including a subset of imagesshowing the light reflected from the floor or obstacle; a microprocessoror equivalent processing unit configured to subtract or difference pairsof images to extract the reflected light and to add or otherwise combinetwo or more pairs of images after subtraction to average out andsuppress the background. With this technique, obstacle detection may beimplemented while the robot is in motion without the need to stop.

The light source can be any of a number of devices including laser andlight emitting diodes. The beam of light generated by the source isgenerally pulsed in a repetitious on and off sequence or in accordancewith a pseudo-random code. The structured light projected by the lightsource on the path before the robot may include focused points of lightor lines of light arrayed horizontally, vertically, or both. Based onthe location of the reflected light in the combined images, theprocessing unit can use triangulation, for example, to determine theposition and/or height of the reflection and distinguish between thefloor and an obstacle in the robot's path. In some embodiments, theobstacle detector further includes a tilt sensor configured to determinethe orientation of the robots to further enhance the accuracy of theobstacle detector.

In some embodiments, the invention features a method for detecting anobstacle, comprising: repeatedly pulsing at least one light source onand off in accordance with a determined sequence; capturing a pluralityof images of light reflected from the path of the robot; generating twoor more difference images by subtracting pairs of the plurality ofimages where each pair of images comprising a first image with the atleast one light source on and a second image with the light source off;combining two or more difference images to enhance the reflected lightrelative to the background; and detecting the obstacle in the path ofthe robot based on a location of the reflected light in the combineddifference images. The method may also determine the location of thereflected light in the combined difference images is determined bytriangulation.

In some additional embodiments, the invention features an obstacledetector comprising: at least one light source configured to projectlight pulsed in a determined sequence; a visual sensor for capturing aplurality of images comprising reflections of the projected light; aprocessing unit configured to subtract pairs of images captured with theat least one light source to extract the reflections, each paircomprising a first image captured with the at least one light source onand a second image captured with the at least one light source off; atilt sensor configured to determine an orientation of the robot; and anobstacle detection unit configured to detect an obstacle in the path ofthe robot based on the determined orientation and the reflections in thepairs of images after subtraction.

BRIEF DESCRIPTION OF THE DRAWINGS

Features of the system and method will be described with reference tothe drawing summarized below. This drawing (not to scale) and theassociated description are provided to illustrate embodiments of thesystem and method and are not intended to limit the scope of the systemor the system or method.

FIG. 1 is a block diagram illustrating one embodiment of the systempresent invention.

FIG. 2 is a schematic representation of one embodiment of a robot systemof the present invention;

FIG. 3 is a block diagram illustrating one embodiment of the systempresent invention;

FIG. 4 is a schematic representation of one embodiment of the robotsystem of the present invention; and

FIG. 5 is a block diagram illustrating one embodiment of the systempresent invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 illustrates a block diagram for one embodiment of the invention.The system includes a visual sensor 100 optionally equipped with anarrow bandpass interference filter matching the frequency of the lasers108. A synchronization circuit 104 receives a synchronization signalfrom a visual sensor 100 and sequentially pulses a laser 108 on and off.Optionally, the synchronization circuit 104 includes a pseudo-randomsequence generator 106. A stream of images 102 acquired from the visualsensor 100 is sent to a persistence model module 110, which combines theinformation from the stream of images 102. A pixel detection module 112determines which pixels in the images 102 are illuminated by light fromthe laser 108, namely the light reflected from the path of the robot. Anobstacle detection module 114 then decides whether the illuminatedpixels correspond to one or more obstacles or do not correspond to anobstacle. For example, in one embodiment, if the obstacle detectionmodule 114 determines that an illuminated pixel corresponds to a floorsurface, then the obstacle detection module determines that theilluminated pixel does not correspond to an obstacle. An optionaltriangulation module 116 computes the 3-D position of the obstacle pointwith respect to a reference frame associated with a device, such as arobot, that carries the visual sensor 100. The synchronization circuit104, the persistence model 110 the pixel detection module 112, theobstacle detection module 114, and the triangulation module 116 arediscussed in greater detail below.

FIG. 3 illustrates a block diagram for another embodiment of theinvention. The robot preferably includes a visual sensor 300 optionallyequipped with narrow bandpass interference filter matching to thefrequency of the lasers. A synchronization circuit 304 receives asynchronization signal from a visual sensor 300 and alternately pulses alaser 308 on and off. Optionally, the synchronization circuit 304includes a pseudo-random sequence generator 306. A stream of images 302acquired from the visual sensor 300 is sent to a tilt compensationmodule 301 which applies the information from the tilt sensor 320 to thestream of images. The tilt-compensated stream of images is thanforwarded to the persistence model 310, which combines the informationfrom the stream of images. A pixel detection module 312 determines whichpixels in the images 302 are illuminated by light from the laser 308. Anobstacle detection module 314 then decides whether or not theilluminated pixels correspond to one or more obstacles. An optionaltriangulation module 316 computes the 3-D position of the obstacle pointwith respect to a reference frame associated with a device, such as arobot, that carries the visual sensor 300. The synchronization circuit304, the tilt compensation module 301, the persistence model 310, thepixel detection module 312, the obstacle detection module 314 and thetriangulation module 316 are discussed in greater detail below.

FIG. 5 illustrates a block diagram for another embodiment of theinvention. The robot preferably includes a visual sensor 500 optionallyequipped with narrow bandpass interference filter matching to thefrequency of the lasers. A synchronization circuit 504 receives asynchronization signal from a visual sensor 500 and alternately pulsesthe lasers 508 and 509 on and off. Optionally, the synchronizationcircuit 504 includes a pseudo-random sequence generator 506. A stream ofimages 502 acquired from the visual sensor 500 is sent to thepersistence model 510, which combines the information from the stream ofimages. A pixel detection module 512 determines which pixels in theimages 502 are illuminated by light from the lasers 508 and 509. Thetilt compensation module 511 determines if the robot system is tiltedbased on the known geometry of lasers 508, 509, and the visual sensor500, and the tilt sensor 520. An obstacle detection module 514 thendecides whether or not the illuminated pixels correspond to one or moreobstacles. For example, in one embodiment, if the obstacle detectionmodule 514 determines that an illuminated pixel corresponds to a floorsurface, then the obstacle detection module determines that theilluminated pixel does not correspond to an obstacle. An optionaltriangulation module 516 computes the 3-D position of the obstacle pointwith respect to a reference frame associated with a device, such as arobot, that carries the visual sensor 500. The synchronization circuit504, the tilt compensation module 511, the persistence model 510, thepixel detection module 512, the obstacle detection module 514 and thetriangulation module 516 are discussed in greater below.

System Tilt Estimation and Compensation

In order for the obstacle detection module to decide if the illuminatedpixels correspond to the floor surface or not it needs to know thesystem position and orientation relative to the floor surface. Thesystem position (height from the floor and initial orientation) areusually fixed at the time of design of mobile platform. However duringoperation, the mobile platform may tilt when passing over a floorsurface that is warped or includes small objects thereon. The tilt ofthe robot causes a shift in and rotation of the theoretical area of theimage corresponding to the illuminated pixels on the floor surface.Without knowing the system tilt, a false positive detection of anobstacle is more likely.

One embodiment of the tilt estimation illustrated in FIG. 2 comprises adedicated tilt sensor 210 (encoder or accelerometer based, for example)which provides the orientation angle relative to the vector of earthgravity to the tilt compensation module. Another embodiment illustratedon FIG. 4 utilizes two parallel laser line segments or strips 405 and406 projected on the floor surface 401 at a different angle. Byanalyzing the distance and angle between the two strips in the imageit's possible to estimate the tilt of the system. This method may bemore economical but can result in a false tilt detection where theobstacle is in a crowded environment passing over an uneven floorsurface.

Synchronization Circuitry

In order to effectively build up a persistence model of the environmentbeing scanned, the laser light must be alternately turned off and on insync with the camera's electronic shutter, and the information as towhether the laser light was on or off for each acquired image frame mustbe passed to the persistence model. Depending on the actual hardwareused, there are several methods to achieve synchronization. For example,some cameras come equipped with a sync in and sync out signal, whichenable, respectively, the camera to receive a synchronization signal tocontrol its electronic shutter, or to provide other devices with a syncsignal. Even if such signals are not available, as may be the case withless expensive cameras, such as USB webcams, there is still within thecamera control circuitry a pin which is the vertical sync signal, fromwhich a sync signal can be derived.

One embodiment of the synchronization circuitry comprises amicrocontroller which receives a sync signal from the camera, counts thesync pulses, and outputs a control signal to a power mosfet to pulse thelaser on for the appropriate amount of time on every even frame.Alternatively, a more sophisticated sequence of laser on/off pulses canbe generated by the microcontroller to reduce the interference fromextraneous periodic sources (as will be described in more detail below).

There are also various ways that the synchronization circuitry cancommunicate to the persistence model whether or not the laser was on oroff for each acquired frame. One method involves indicating the state ofthe laser on a digital i/o line which the CPU running the persistencemodel can read. Another method comprises the synchronizationmicrocontroller simulating the pressing of the ‘snapshot’ button of thecamera (if one exists, such as on the Logitech 4000 webcam) whenever thelaser is on, and this information is passed to the CPU running thepersistence model as part of the camera image data.

Persistence Modeling of Frames

A persistence model of frames is a linear combination of images thatattempts to maximize the correlated persistence of a laser on an object,but minimize uncorrelated background movement artifacts. A set of nframes can be denoted by {f₁, f₂, . . . , f_(n)}.

In one embodiment, a frame f₁ is represented by a matrix of grayscalepixel values. The dimensions of the matrix correspond to the dimensions,in pixels, of the image received from a visual sensor. The visual sensorcan receive multiple channels of data, such as RGB color data, and use amodel or math filter to convert the multiple-channel image to asingle-channel image, such as a grayscale image. In one embodiment, themodel or filter attempts to extract the “laser on” pixels from the“laser off” pixels. If multiple channels are used, the filter can beused to differentiate among several laser wavelengths, even if multiplewavelengths appear in the same image. For example, in oneimplementation, a first laser wavelength is used to detect obstaclesthat are relatively low to the floor surface, and a second laserwavelength is used to detect obstacles that are relatively high off thefloor surface. In one embodiment, the filter performs a weightedaddition of the channels, wherein each channel's weighting isproportional to the relative intensity of the laser's wavelength asdetected by the channel. One embodiment of the filter performs a medianfilter using some window size, such as 5 pixels by 5 pixels or 7 pixelsby 7 pixels. More complex filters can be used to account for thepoint-spread pattern of the laser, which may, for example, correspond toa dot, a line, or a Gaussian distribution. The resulting filtered framecorresponds to the intensity of the filtered laser light in the image.

A persistence model d_(n) associated with the set of frames {f₁, f₂, . .. , f_(n)} can be expressed in the form given by Equation 1.

$\begin{matrix}{d_{n} = {\sum\limits_{i = 1}^{n}\; {c_{i}f_{i}}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

In Equation 1, the constants c₁, c₂, . . . , c_(n) are called the“persistence scalers” associated with the frames f₁, f₂, . . . , f_(n).

One problem with frame differencing alone (that may exist in the priorart) is that even small motion can cause very bright artifacts near theedge of high contrast object. These artifacts are difficult todistinguish from the structured laser light. One possible method toavoid this is to model not just the instantaneous difference betweenframes, f_(2n)-f_(2n-1), but rather a more complex combination of pastframes f_(i), as given in Equation 2.

$\begin{matrix}{d_{2\; n} = {\sum\limits_{i = 1}^{n}\; ( {{a_{2\; i}f_{2\; i}} - {a_{{2\; i} - 1}f_{{2\; i} - 1}}} )}} & {{Equation}\mspace{14mu} 2}\end{matrix}$

Different persistence models can be created using Equation 2 by choosingdifferent values for the constants a₁, a₂, . . . , a_(2n). Each constanta; is known as a “frame difference persistence scaler” associated withthe frame f_(i).

One example of a model for persistence is an exponential decay model.This model is based on the idea that for most cases, the laser projectedonto an object will remain in approximately the same place for a timescale of a certain number of frames. At the same time, the probabilityof a high-contrast edge motion artifact occupying the same pixel regionover multiple difference frames (other than oscillation, which isdiscussed in greater detail below in the context of periodic error) isrelatively unlikely and should decrease with time.

In one embodiment, the model assumes independence of persistenceprobabilities between frames for a given occupied pixel region. For eachdifference frame, a pixel region being illuminated by laser lightreflected by an obstacle has probability p of having been illuminated inthe previous difference frame. The assumption of implies that theprobability of the region being illuminated for k consecutive differenceframes will then be p^(k). Then, given that a pixel region isilluminated in the nth difference frame, the probability that the pixelregion is also illuminated in the nth difference frame, where i<n, isgiven by the expression shown in Equation 3.

P(r _(i) |r _(n))=p ^(n-i)  Equation 3

Equation 3 implies that a_(i)=p^(n-i) can be used as the framedifference persistence scalers, and, thus, the frame differencingrelationship discussed earlier in connection with Equation 2 can bewritten as follows in Equation 4.

$\begin{matrix}{d_{2\; n} = {\sum\limits_{i = 1}^{n}\; ( {{p^{{2\; n} - {2\; i}}f_{2\; i}} - {p^{{2\; n} - {2\; i} + 1}f_{{2\; i} - 1}}} )}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

In one embodiment, the choice of p depends on the persistenceprobability of real world obstacles and should be determined throughexperimentation, such as robot navigation testing with a laser andlikely obstacles. It is possible, however, to estimate a relativelylikely bound on p through experimentation. The bound on p can, forexample depend on characteristics of the system such as the robot'stypical velocity, the width of the laser, and the resolution of thevisual sensor.

One benefits of the exponential decay model is its relativecomputational simplicity. It is not only a relatively simple probabilitymodel, but it can also be computed using a relatively fast inductivealgorithm. Advantageously, exponentials need not be computed, andfloating point math need not be used. These features represent animportant design advantage for streaming video and real-time obstacledetection applications.

Equation 4 can be rewritten in the following form.

$\begin{matrix}\begin{matrix}{d_{2\; n} = {f_{2\; n} - {pf}_{2\; n} + {f^{2}( {{\sum\limits_{i = 1}^{n - 1}\; {p^{{2\; n} - {2\; i}}f_{2\; i}}} - {\sum\limits_{i = 1}^{n - 1}\; {p^{{2\; n} - {2\; i} + 1}f_{{2\; i} - 1}}}} )}}} \\{= {f_{2\; n} - {p( {f_{{2\; n} - 1} - {pd}_{{2\; n} - 2}} )}}}\end{matrix} & {{Equation}\mspace{14mu} 5}\end{matrix}$

If p is taken to be a rational number that can be expressed in the formp=m/2^(N), where m and N are integers, the iterative algorithm shown inEquation 5 can be computed relatively quickly, using only integer math(multiplies and shifts).

Periodic Noise Reduction

One potential source of false frame differencing signals is periodicsources of light. Because most CCDs operate at 30 frames per second,frame differencing will be sensitive to light sources that flicker at 15Hz or at a whole-number multiple of 15 Hz, such as 30 Hz, 45 Hz, 60 Hz,and so forth. Computer monitors, TV remote controls, and electric lightsare examples of devices that flicker at these rates and therefore maycause problems. For example, the system may mistakenly identify signalsas laser signals. Periodic patterns combined with constant devicemotion, as experienced when a device moves along a floor that has aregular pattern of tiles, can also cause periodic error.

One way to filter out periodic error is to use a more complex pattern ofon/off frames rather than the standard even/odd method. One embodimentemploys a long bit sequence that determines if a particular frame shouldbe added or subtracted. One example of an appropriate bit sequence is asequence formed by repeating a sub-sequence multiple times. For example,a sub-sequence {b₁, b₂, . . . , b_(m)} that is m bits long can berepeated to form a longer sequence {b₁, b₂, . . . , b_(m), b₁, b₂, . . ., b_(m), b₁, b₂, . . . , b_(m), . . . }. It will be understood that thesub-sequence can be repeated as necessary until each frame of interestis associated with a sequence element. Thus, because the sequencerepeats every m terms, the terms of the sub-sequence, denoted herein by{b_(i mod m)}, constitute a complete sub-sequence for i=1, 2, . . . , m.In the example illustrated earlier in connection with Equation 2,Equation 3, Equation 4, and Equation 5, the sub-sequence {b_(i mod m)}corresponds to {b_(i mod 2) }={1, 2}. However, other appropriatesub-sequences sequences will be readily determined by one of ordinaryskill in the art. Then, d _(n) can be computed according to the Equation6.

$\begin{matrix}{d_{n} = {\sum\limits_{i = 1}^{n}\; {( {- 1} )^{b_{i\mspace{11mu} {mod}\mspace{11mu} m}}p^{n - i}f_{i}}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$

Detecting Illuminated Pixels

Given the current persistence model d_(n), several methods can be usedto determine the pixel locations at which the laser light is detected.In one embodiment, a decision can be made on a pixel-by-pixel basis inthe following manner. Each pixel value is compared to a threshold, andif the pixel value exceeds the threshold, the laser light is deemed tobe present at the location represented by the corresponding pixel. Anappropriate threshold can be selected to maximize the probability ofrecognizing the laser correctly, and an appropriate threshold can dependon the robot's typical velocity, ambient lighting conditions,reflectivity of the observed surfaces, and the effectiveness of thefilter that filters images from the visual sensor. In anotherembodiment, the persistence model is analyzed on a block-by-block basis.An appropriate block size can be selected using characteristics of thesystem such as the visual sensor resolution and the desired resolutionof obstacle detection. This method tends to increase the method'srobustness to outliers. Yet another embodiment considers the epipolarconstraint of the beam of light. For each epipolar ray, the methodselects, at most, one pixel that is most likely to be illuminated. Thisembodiment is based on the fact that, if transparent objects aredisregarded, a thin beam of light illuminates only one location. Thus,within the locus of possible beam locations (the epipolar ray), thereis, at most, only one location at which the beam is actually detected.

Detecting Obstacles

For each image coordinate at which the light pattern is detected, onecan determine whether or not it corresponds to an obstacle. Thiscomputation can be done in several ways. One embodiment uses an accuratecalibration of the relative poses of the imager and the light source andthen triangulates the observed image coordinates to compute a positionin 3-D space. In one implementation, these positions from multipleframes in sequence are used to reconstruct obstacles in a more robustfashion. Another embodiment avoids the requirement of an accuraterelative pose calibration, and, rather than estimating the 3-D positionof the detected image point, distinguishes whether or not the pointcorresponds to an obstacle. This section describes how such computationsare performed for an embodiment in which the structured light is a laserstripe.

An observed light point can be classified as either corresponding to anobstacle or not corresponding to an obstacle. This classification can bedone based at least in part on the image coordinates associated with theobserved light point. Points that correspond to the floor plane liewithin a particular region of the image, and points that correspond toan object at a height higher or lower than the floor plane will lie inother regions of the image. For example, if the light source is a laserstripe, and the laser stripe is observed on the floor, then it will beobserved on the image as a particular line segment. If the mobile devicemoves so that laser stripe no longer hits the floor but rather anobstacle, then the laser light is detected in the image no longer on thefloor's line segment, but rather above it or below it. Whether thedetection occurs above or below the floor line depends on theconfiguration of the visual sensor and light source. Thus, to determinewhether or not an object is present, it is sufficient to determinewhether or not the detected light point corresponds to the floor linesegment or not. If one also has information relating to the relativeposition of the imager and light source, one can further determinewhether the obstacle is above or below the floor plane. Examples ofobstacles that can lie below the floor plane include depressions, holes,stairs, and the like.

An obstacle is defined as an observed light point not corresponding tothe floor plane. However in case of the robot tilt over small objects onthe floor (like wires, books, small toys, doorways) the region of theimage corresponding to the floor plane will shift and/or rotate. It cancause an undesirable false-positive detection of an obstacle. Oneembodiment uses a tilt sensor to measure the robot pose to re-adjust theregion of the image corresponding to the floor plane.

If an accurate calibration of the relative pose of the imager and lightsource is known, one can compute the 3-D position in space of theobstacle with respect to the mobile device. This can provide additionalfunctionality, namely, one can decide whether the object is large enoughto be of concern. In particular, if the object is very small, it can besafe to move over it. It can also be possible to maintain a 3-D map (or2-D map by projecting onto the floor plane) of the locations ofobstacles, so that the mobile device reacts not only to detections fromthe current image, but also to information from the past. One way tocompute a 3-D location corresponding to the detected image point is touse the procedure discussed below in connection with Equation 7 andEquation 8.

$\begin{matrix}{x_{p} = {\frac{1}{f}( {x_{d} - t} )}} & {{Equation}\mspace{14mu} 7}\end{matrix}$

In Equation 7, x_(d) corresponds to the coordinates of the detectedimage point, and x_(p) corresponds to the corresponding coordinates ofthe detected image point in the ideal perspective projection plane.x_(p) can be obtained from x_(d) by using the focal length f and theoptical axis vector t of the imager. In one embodiment, x_(d), x_(p),and t are represented by 2-element vectors that correspond tocoordinates in an (x, y) plane.

More sophisticated formulas can be used to take into account lensdistortion factors, which can be necessary to incorporate for imageswith lenses with significant distortion, such as a wide angle lens. Forexample, if x is represented with the homogeneous coordinates of theperspective projection point x_(p), (i.e., make x_(p) a 3-dimensionalvector by appending a 1 to it as a third coordinate), then x representsthe line from the imager focal point to the 3-D location of the detectedobstacle. The intersection of this line with the plane of lightemanating from the light source determines the 3-D location of theobstacle. The intersection point, X, of x with the light plane iscomputed using Equation 8.

$\begin{matrix}{X = {\frac{p_{i} \cdot n_{i}}{x \cdot n_{i}}x}} & {{Equation}\mspace{14mu} 8}\end{matrix}$

In Equation 8, n_(i), corresponds to the 3-D vector defining the normalto the laser light plane, and p_(i), corresponds to a point on the laserlight plane, defined with respect to the visual sensor's referenceframe. The symbol “•” denotes the vector dot product operation. X isthus the 3-D coordinate of the detected obstacle in the visual sensor'sreference frame. If the coordinate with respect to the mobile device'sreference frame is desired, it can be computed by applying theappropriate coordinate transformation, assuming that the visual sensor'sposition relative to the mobile device's reference frame is known.

Covering Entire Device Profile

With a single laser stripe, it can be relatively difficult to guaranteedetection of all objects that may collide with a mobile device of agiven height. For example, a single laser stripe aimed at a floorsurface should enable the detection of an obstacle near the ground, butit is relatively unlikely to detect an overhanging obstacle, such as aseat of a chair or a tabletop of a coffee table. To cover the fullforward-facing profile of the mobile device, multiple laser stripes canbe used.

FIG. 2 illustrates a configuration of the system in which two laserstripes can provide coverage of the full forward-facing profile of amobile device. A robot 200 is supported by a floor surface 201. A visualsensor 202 having a field of view 209 is coupled to the robot 200. Therobot 200 is equipped with a first stripe-producing laser 203 and asecond stripe-producing laser 204. The first laser 203 is locatedapproximately at the top of the robot 200 and emits a first beam 205that is directed in a generally downward direction. The second laser 204is located approximately at the bottom of the robot 100 and emits asecond beam 206 directed in a generally upward direction. When the firstbeam 205 and/or the second beam 206 intersects an obstacle that lieswithin the field of view 209 of the visual sensor 202, the robot 200 candetect the obstacle. As FIG. 1 illustrates, the robot 200 can detect afirst obstacle 207 that lies in the path of the top portion of the robot200 because the beam 206 intersects the obstacle 207 within the field ofview 209 of the visual sensor 202. Furthermore, the robot can detect asecond obstacle 208 that lies in the path of the bottom portion of therobot 200 because the beam 205 intersects the obstacle 208 within thefield of view 209 of the visual sensor 202. In this fashion, the robot200 obtains obstacle detection covering its full forward-facing profile.

Various embodiments of the system and method have been described above.Although this system and method have been described with reference tothese specific embodiments, the descriptions are intended to beillustrative of the system and method and are not intended to belimiting. Various modifications and applications may occur to thoseskilled in the art without departing from the true spirit and scope ofthe system or the method.

What is claimed is:
 1. A mobile robot, the mobile robot comprising: atleast one laser light source configured to at least: project a firstlaser stripe ahead of the mobile robot, and project a second laserstripe ahead of the mobile robot, wherein the first laser stripe and thesecond laser stripe cover an entire profile of the mobile robot; avisual sensor configured to detect the first laser stripe and the secondlaser stripe; an obstacle detection unit configured to detect, fromimages captured by the visual sensor, an obstacle based at least in parton detection of an intersection of the first laser stripe, or the secondlaser stripe, or both the first laser stripe and the second laserstripe, with the obstacle; and a processing unit configured to maintaina two dimensional map of detected obstacles.
 2. The mobile robot asdefined in claim 1, wherein: the processing unit is further configuredto maintain a three dimensional map of an area traversed by the mobilerobot, the three dimensional map including a three dimensional mappingin space of one or more of the detected obstacles; the at least onelaser light source comprises a first laser light source and a secondlaser light source, and the first laser light source is configured todownwardly project the first laser stripe, and the second laser lightsource is configured to upwardly project the second laser stripe,wherein the first laser stripe and the second laser stripe cover a fullforward facing profile of the mobile robot, and the obstacle detectionunit is configured to detect an overhanging obstacle based at least inpart on an intersection of the upwardly projected second laser stripewith the overhanding obstacle; and the visual sensor comprises abandpass interference filter.
 3. The mobile robot as defined in claim 1,wherein the processing unit is further configured to maintain a threedimensional map of an area traversed by the mobile robot, the threedimensional map including a three dimensional mapping in space of thedetected obstacles.
 4. The mobile robot as defined in claim 1, whereinthe at least one laser light source is configured to: downwardly projectthe first laser stripe, and upwardly project the second laser stripe,wherein the first laser stripe and the second laser stripe cover a fullforward facing profile of the mobile robot.
 5. The mobile robot asdefined in claim 1, wherein the visual sensor is configured to cover thefull forward facing profile of the mobile robot.
 6. The mobile robot asdefined in claim 1, wherein the at least one laser light source isconfigured to upwardly project the first laser stripe, and the obstacledetection unit is configured to detect an overhanging obstacle based atleast in part on an intersection of the upwardly projected first laserstripe with the overhanding obstacle.
 7. The mobile robot as defined inclaim 1, wherein the obstacle detection unit is configured to detect asize of the obstacle based at least in part on an intersection of boththe first laser stripe and the second laser stripe with the obstacle. 8.The mobile robot as defined in claim 1, wherein the at least one laserlight source is configured to: project the first laser stripe onto afloor surface a first distance ahead of a current position of the mobilerobot; and project the second laser stripe onto the floor surface asecond distance ahead of the current position of the mobile robot, thesecond distance greater than the first distance.
 9. The mobile robot asdefined in claim 1, wherein the visual sensor comprises a single camerautilized for obstacle detection and at least one of object recognition,object tracking, video streaming for surveillance, tele-operation, orvideo-conferencing.
 10. The mobile robot as defined in claim 1, whereinthe obstacle detection unit configured to detect, from images capturedby the visual sensor, an obstacle based at least in part on adetermination that a position of the first stripe in an image capturedby the visual sensor is at a height higher than a floor plane of a flooron which the mobile robot is supported.
 11. A method of detectingobstacles in the path of a mobile robot, the method comprising:projecting, by a light source affixed to the mobile robot, a firststripe ahead of the mobile robot, and projecting, by the light sourceaffixed to the mobile robot, a second stripe ahead of the mobile robot,wherein the first stripe and the second stripe cover an entire profileof the mobile robot; capturing, by a sensor, one or more images of thefirst stripe and the second stripe; detecting, by a processing unit,from one or more images captured by the visual sensor, an obstacle basedat least in part on detection of an intersection of the first stripe, orthe second stripe, or both the first stripe and the second stripe, withthe obstacle; and maintaining a map of detected obstacles.
 12. Themethod as defined in claim 11, wherein projecting, by a light sourceaffixed to the mobile robot, the first stripe ahead of the mobile robot,further comprises downwardly projecting the first stripe using a firstdevice; wherein projecting, by a light source affixed to the mobilerobot, the second stripe ahead of the mobile robot, further comprisesupwardly projecting the second stripe using a second device; the methodfurther comprising: detecting, by the processing unit, an overhangingobstacle based at least in part on an intersection of the upwardlyprojected second stripe with the overhanding obstacle; and maintaining,by the processing unit, a three dimensional map of an area traversed bythe mobile robot, the three dimensional map including a threedimensional mapping in space of one or more of the detected obstacles;13. The method as defined in claim 11, the method further comprisingmaintaining a three dimensional map of an area traversed by the mobilerobot, the three dimensional map including a three dimensional mappingin space of the detected obstacles.
 14. The method as defined in claim11, the method further comprising: downwardly projecting the firststripe, and upwardly projecting the second stripe, wherein the firststripe and the second stripe cover a full forward facing profile of themobile robot.
 15. The method as defined in claim 11, wherein the visualsensor is configured to cover the full forward facing profile of themobile robot.
 16. The method as defined in claim 11, the method furthercomprising: upwardly projecting the first stripe; and detecting anoverhanging obstacle based at least in part on an intersection of theupwardly projected first stripe with the overhanding obstacle.
 17. Themethod as defined in claim 11, the method further comprising detecting asize of the obstacle based at least in part on an intersection of boththe first stripe and the second stripe with the obstacle.
 18. The methodas defined in claim 11, the method further comprising: projecting thefirst stripe onto a floor surface a first distance ahead of a currentposition of the mobile robot; and projecting the second stripe onto thefloor surface a second distance ahead of the current position of themobile robot, the second distance greater than the first distance. 19.The method as defined in claim 11, wherein the visual sensor comprises asingle camera, the method further comprising utilizing the single camerafor at least one of video streaming for surveillance, tele-operation, orvideo-conferencing, in addition to utilizing the visual sensor forobstacle detection.
 20. The method as defined in claim 11, the methodfurther comprising detecting, from images captured by the visual sensor,an obstacle based at least in part on a determination that a position ofthe first stripe in an image captured by the visual sensor is at aheight higher than a floor plane of a floor on which the mobile robot issupported.